Image processing apparatus, control method therefor, and program

ABSTRACT

An image processing apparatus which corrects input image data on the basis of the image data and foreign substance information including information associated with a position and a size of a foreign substance adhering near an image sensor in an image capturing apparatus which has captured the image data includes a correction unit which corrects the image data on the basis of the image data and the foreign substance information so as to reduce an influence of a shadow of the foreign substance cast in the image data, a uniformity determination unit which determines uniformity of an image in a region surrounding the foreign substance in the image data, and a control unit which, when the uniformity determination unit determines that the uniformity is lower than a predetermined value, prevents the correction unit from correcting the image data.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technique of suppressing degradationof image quality due to foreign substances adhering to the surface of anoptical low-pass filter or the like in an image capturing apparatuswhich uses an image sensor such as a CCD or a CMOS sensor.

2. Description of the Related Art

Recently, many kinds of image capturing apparatuses which generate imagesignals by using image sensors such as CCDs and record them as data, forexample, digital cameras and digital video cameras, have come to be onthe market. Digital cameras need not use any photosensitive films, whichhave been used as recording media, and record images as data on datarecording media such as semiconductor memory cards and hard disk drivesare used in place of such films. These data recording media allow writeand erase operations many times, and hence can save expenses forconsumables. That is, such media are very convenient.

In general, a digital camera is equipped with an LCD (Liquid CrystalDisplay) monitor device capable of displaying captured images as neededand a detachable large-capacity storage device.

Using a digital camera comprising these two devices makes it unnecessaryto use any films as recording media which have been used as consumablesand allows immediate checking of captured images by displaying them onthe LCD monitor device. This makes it possible to erase unsatisfactoryimage data on the site or recapture the same image as needed. That is,this camera has remarkably improved the photo capturing efficiency ascompared with a silver halide camera using films.

Such convenience and technical innovations such as an increase in thenumber of pixels of an image sensor have widened the application rangeof digital cameras. Recently, many lens-interchangeable digital camerassuch as single-lens reflex cameras have been available.

Foreign substances such as dust and foreign particles (to be simplyreferred to as foreign substances hereinafter) sometimes adhere to thesurfaces of the image sensor protective glass fixed to the image sensorand of an optical filter and the like (to be collectively referred to asimage sensor optical system components hereinafter) arranged near theimage sensor. When a foreign substance adheres to an image sensoroptical system component in this manner, the foreign substance blockslight. In this case, for example, an object image corresponding to thatportion cannot be obtained, resulting in degradation of the imagequality of the captured image.

Not only a digital camera but also a camera using silver halide filmssuffers the problem that a foreign substance on a film is reflected in aframe. In a camera using films, however, since a film moves frame byframe, the same foreign substance is rarely reflected in all the frames.

In contrast to this, the image sensor of a digital camera does not move.That is, the camera performs image capturing by using the same imagesensor. If, therefore, a foreign substance adheres to an image sensoroptical system component, the same foreign substance is reflected inmany frames (captured images). A lens-interchangeable digital camera, inparticular, suffers a problem that a foreign substance tends to enterthe camera when the user interchanges lenses.

The user therefore must always pay attention to the adhesion of aforeign substance to an image sensor optical system component, andspends much effort to check and clean foreign substances. Since an imagesensor is placed relatively deep in the camera, in particular, it is noteasy to check and clean foreign substances.

A foreign substance readily enters the lens-interchangeable digitalcamera during lens attaching/detaching operation. In addition, most oflens-interchangeable digital cameras have a focal-plane shutter in frontof the image sensor, and a foreign substance easily adheres to an imagesensor optical system component.

A foreign substance on such an image sensor usually adheres to theprotective glass or the optical filter instead of the surface of theimage sensor, and hence the image forming state of the foreign substancevaries depending on the aperture value or pupil position of thephotographing lens. That is, as the aperture value becomes closer to thefull open aperture value, a foreign substance blurs. Even if a smallforeign substance adheres to such a component, there is almost noinfluence on the captured image. In contrast, as the aperture valueincreases, the foreign substance is clearly focused to influence thecaptured image.

There is known a method of making a foreign substance less noticeable byusing a combination of a normally captured image and an image,comprising only a foreign substance on the image sensor, which isprepared in advance by capturing a white wall or the like while the lensis stopped down (see Japanese Patent Laid-Open No. 2004-222231). Morespecifically, a region corresponding to an image in which a foreignsubstance is reflected is interpolated with surrounding pixels.

Assume that a given pixel is to be interpolated by surrounding pixels.In this case, if an image area around the interpolation target pixel isnot uniform, the user can easily-sense discontinuity between theinterpolated region and its surrounding region.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the aboveproblem, and has as its object to properly correct a captured image andsuppress the influence of a foreign substance on the captured image evenwhen the foreign substance adheres to a protective glass fixed to animage sensor, or an optical filter arranged near the image sensor.

In order to solve the above problem and achieve the above object,according to a first aspect of the present invention, there is providedan image processing apparatus which corrects input image data on thebasis of the image data and foreign substance information includinginformation associated with a position and a size of a foreign substanceadhering near an image sensor in an image capturing apparatus which hascaptured the image data, comprising a correction unit adapted to correctthe image data on the basis of the image data and the foreign substanceinformation so as to reduce an influence of a shadow of the foreignsubstance cast in the image data, a uniformity determination unitadapted to determine uniformity of an image in a region surrounding theforeign substance in the image data, and a control unit adapted to, whenthe uniformity determination unit determines that the uniformity islower than a predetermined value, prevent the correction unit fromcorrecting the image data.

According to a second aspect of the present invention, there is provideda method of controlling an image processing apparatus which correctsinput image data on the basis of the image data and foreign substanceinformation including information associated with a position and a sizeof a foreign substance adhering near an image sensor in an imagecapturing apparatus which has captured the image data, comprising thesteps of correcting the image data on the basis of the image data andthe foreign substance information so as to reduce an influence of ashadow of the foreign substance cast in the image data, determininguniformity of an image in a region surrounding the foreign substance inthe image data, and when it is determined in the uniformity determiningstep that the uniformity is lower than a predetermined value, preventingcorrection of the image data in the correcting step.

According to a third aspect of the present invention, there is provideda program for controlling an image processing apparatus which correctsinput image data on the basis of the image data and foreign substanceinformation including information associated with a position and a sizeof a foreign substance adhering near an image sensor in an imagecapturing apparatus which has captured the image data, causing acomputer to execute the steps of correcting the image data on the basisof the image data and the foreign substance information so as to reducean influence of a shadow of the foreign substance cast in the imagedata, determining uniformity of an image in a region surrounding theforeign substance in the image data, and when it is determined in theuniformity determining step that the uniformity is lower than apredetermined value, preventing correction of the image data in thecorrecting step.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the circuit arrangement of alens-interchangeable single-lens reflex digital camera as an imagecapturing apparatus according to the first embodiment of the presentinvention;

FIG. 2 is a perspective view showing the outer appearance of a digitalcamera according to the first embodiment;

FIG. 3 is a vertical sectional view showing the internal structure ofthe digital camera according to the first embodiment;

FIG. 4 is a flowchart for explaining foreign substance detectionprocessing in the digital camera according to the first embodiment;

FIG. 5 is a view showing an example of the data format of foreignsubstance correction data;

FIG. 6 is a flowchart for explaining the details of the foreignsubstance region acquisition routine performed in step S27 in FIG. 4;

FIG. 7 is a view showing the unit of processing in the foreign substanceregion determination processing performed in step S62 in FIG. 6;

FIG. 8 is a view showing an outline of foreign substance region sizecalculation performed in step S63 in FIG. 6;

FIG. 9 is a flowchart for explaining the details of the image capturingprocessing routine performed in step S24 in FIG. 4;

FIG. 10 is a view schematically showing the system configuration of animage processing apparatus;

FIG. 11 is a view showing an example of a GUI in the image processingapparatus;

FIG. 12 is a view showing the internal structure of an image editingprogram;

FIG. 13 is a view showing the data structure of an editing log in theimage editing program;

FIG. 14 is a flowchart for explaining a basic procedure of an automaticrepair process;

FIG. 15 is a flowchart showing a procedure of an automatic repairprocess in the first embodiment;

FIG. 16 is a flowchart for explaining the details of an interpolationroutine;

FIG. 17 is a flowchart showing a procedure of an automatic repairprocess in the second embodiment;

FIG. 18 is a flowchart showing a procedure of an automatic repairprocess in the third embodiment;

FIG. 19 is a flowchart showing a uniformity determination valuecalculation processing procedure in the third embodiment;

FIGS. 20A and 20B are views each showing an example of a smoothingfilter;

FIG. 21 is a flowchart showing a uniformity determination valuecalculation processing procedure in the fourth embodiment;

FIG. 22 is a flowchart showing a uniformity determination valuecalculation processing procedure in the fifth embodiment;

FIG. 23 is a flowchart showing a modification of the uniformitydetermination value calculation processing procedure in the fifthembodiment;

FIG. 24 is a view for explaining the details of cancellation processingfor editing processing;

FIG. 25 is a flowchart for explaining an image processing procedure tobe executed after the cancellation of editing processing;

FIG. 26 is a view showing an example of a cancellation designationdialog;

FIGS. 27A to 27D are views for explaining a phenomenon in a case whereinediting processing applied to a region designated as a copy sourceregion is cancelled;

FIG. 28 is a flowchart for explaining the details of cancellationinhibition determination processing;

FIG. 29 is a view showing an example of the data format of foreignsubstance correction data to which a cancellation flag is added;

FIG. 30 is a flowchart showing an automatic repair process procedure ina case wherein a cancellation flag is added; and

FIG. 31 is a view for explaining the details of update processing forthe cancellation flag.

DESCRIPTION OF THE EMBODIMENTS

The preferred embodiments of the present invention will be described indetail below with reference to the accompanying drawings.

First Embodiment

This embodiment will exemplify a case wherein a camera body detects aforeign substance to attach foreign substance correction data to imagedata, and an image processing apparatus outside the camera removes theforeign substance from the image data by using the foreign substancecorrection data attached to the image data.

FIG. 1 is a block diagram showing the circuit arrangement of alens-interchangeable single-lens reflex digital camera as an imagecapturing apparatus according to the first embodiment of the presentinvention.

Referring to FIG. 1, a microcomputer 402 controls the overall operationof the camera, for example, processing for image data output from animage sensor (a CCD in this embodiment) 418 and display control on anLCD monitor device 417.

A switch (SW1) 405 is turned on when the user half-presses a releasebutton 114 (see FIG. 2). When the switch (SW1) 405 is turned on, thedigital camera of this embodiment is set to be ready for imagecapturing. A switch (SW2) 406 is turned on when the release button 114is pressed to the second stroke position (a full-pressed state). Whenthe switch (SW2) 406 is turned on, the digital camera of this embodimentstarts image capturing.

A lens control circuit 407 communicates with a photographing lens 200(see FIG. 3) and performs driving control on the photographing lens 200and aperture vanes.

Referring to FIG. 1, an external display control circuit 408 controls anexternal display device (OLC) 409 and a display device (not shown) inthe viewfinder. A switch sense circuit 410 transfers signals from manyswitches and the like including an electronic dial 411 provided for thecamera to the microcomputer 402.

An electronic flash emission control circuit 412 is grounded through anX contact 412 a and controls an external electronic flash unit. Adistance measurement circuit 413 detects a defocus amount for AF withrespect to an object. A photometry circuit 414 measures the brightnessof an object.

A shutter control circuit 415 controls the shutter to properly exposethe image sensor. An LCD monitor device 417 and a backlight illuminationdevice 416 comprises an image display device. An external storage device419 is, for example, a hard disk drive, semiconductor memory card, orthe like which is detachably mounted in the camera body.

The following components connect to the microcomputer 402: an A/Dconverter 423, an image buffer memory 424, an image processing circuit425 comprising a DSP and the like, and a pixel defect position memory426 storing information indicating that a predetermined pixel in theimage sensor is itself a defect. A foreign substance position memory 427storing a pixel position in the image sensor at which an image failurehas occurred due to a foreign substance also connects to themicrocomputer 402. Note that this apparatus preferably uses nonvolatilememories as the pixel defect position memory 426 and the foreignsubstance position memory 427. The pixel defect position memory 426 andthe foreign substance position memory 427 may store information usingdifferent addresses within the same memory space.

Reference numeral 428 denotes a nonvolatile memory storing programs andthe like which the microcomputer 402 executes.

FIG. 2 is a perspective view showing the outer appearance of the digitalcamera according to this embodiment. FIG. 3 is a vertical sectional viewof FIG. 2.

Referring to FIG. 2, an eyepiece window 111 for the observation of theviewfinder, an AE (Automatic Exposure) lock button 112, a distancemeasurement point selection button 113 for AF, and the release button114 for image capturing are provided on the upper portion of a camerabody 100. The electronic dial 411, a capturing mode selection dial 117,and the external display device 409 are also provided on the upperportion. The electronic dial 411 is a multi-function signal input devicewhich the user operates to input numerical values to the camera,together with other operation buttons, or to switch image capturingmodes. The external display device 409 comprises a liquid crystaldisplay device, and displays capturing conditions such as a shutterspeed, an aperture value, and an image capturing mode, and other piecesof information.

The LCD monitor device 417 which displays captured images, varioussetting windows, and the like, a monitor switch 121 for turning on/offthe LCD monitor device 417, a 4-way selector switch 116, and a menubutton 124 are provided on the rear surface of the camera body 100.

The 4-way selector switch 116 has four buttons arranged vertically andhorizontally and a SET button placed in the center. The user uses thisswitch to issue, to the camera, an instruction to select or execute amenu item displayed on the LCD monitor device 417 or the like.

The menu button 124 is a button for causing the LCD monitor device 417to display a menu window for performing various kinds of settings in thecamera. When, for example, selecting and setting an image capturingmode, the user presses the menu button 124, then selects a desired modeby operating the up, down, left, and right buttons of the 4-way selectorswitch 116, and finally sets the SET button while the desired mode isselected, thereby completing the setting.

Since the LCD monitor device 417 of this embodiment is of a transmissiontype, the user cannot visually check any image only by driving the LCDmonitor device. That is, it is always necessary to provide the backlightillumination device 416 on the rear surface of this monitor device, asshown in FIG. 3. As described above, the LCD monitor device 417 and thebacklight illumination device 416 comprise an image display device.

As shown in FIG. 3, the photographing lens 200 as an image capturingoptical system is detachably mounted on the camera body 100 through alens mount 202. Referring to FIG. 3, reference numeral 201 denotes acapturing optical axis; and 203, a quick return mirror.

The quick return mirror 203 is placed in the capturing optical path, andcan move between a position at which the mirror guides object light fromthe photographing lens 200 to the viewfinder optical system (theposition shown in FIG. 3, which will be referred to as an oblique mountposition) and a position at which the mirror retreats outside thecapturing optical path (which will be referred to as a retreatposition).

Referring to FIG. 3, object light guided from the quick return mirror203 to the viewfinder optical system is formed into an image on afocusing screen 204. Reference numeral 205 denotes a condenser lens forimproving the visibility of the viewfinder; and 206, a pentagonal roofprism, which guides object light passing through the focusing screen 204and the condenser lens 205 to an eyepiece 208 for the observation of theviewfinder and a photometry sensor 207.

Reference numerals 209 and 210 denote rear and front curtainsconstituting a shutter. Opening the rear and front curtains 209 and 210will expose the image sensor 418 as a solid-state image sensor placedbehind them for a necessary period of time. The A/D converter 423, theimage processing circuit 425, and the like process a captured imageconverted into an electrical signal for each pixel by the image sensor.The external storage device 419 stores the resultant data as image data.

The image sensor 418 is held on a printed board 211. A display board 215as another printed board is placed behind the printed board 211. The LCDmonitor device 417 and the backlight illumination device 416 arearranged on the opposite side surface of the display board 215.

Reference numeral 419 denotes an external storage device storing imagedata; and 217, a battery (portable power supply). The external storagedevice 419 and the battery 217 are detachable from the camera body.

(Foreign Substance Detection Processing)

FIG. 4 is a flowchart for explaining foreign substance detectionprocessing (detection processing for a pixel position at which an imagedefect is caused by a foreign substance) in the digital camera accordingto this embodiment. The microcomputer 402 executes this processing byexecuting the foreign substance detection processing program stored inthe memory 428.

Foreign substance detection processing is performed by capturing aforeign substance detection image. When foreign substance detectionprocessing is to be performed, the camera is placed to direct thecapturing optical axis 201 of the photographing lens 200 to a surfacehaving a uniform color such as the exit surface of a surface lightsource unit or a white wall, and preparation is made to capture an imagefor foreign substance detection. Alternatively, the user mounts a lightunit (a compact light source device to be mounted in place of a lens)for foreign substance detection on the lens mount 202 and makespreparation to capture an image for foreign substance detection. As alight source for a light unit, for example, a white LED may be used, andit is preferable to adjust the size of an emission surface to set apredetermined aperture value (for example, F64 in this embodiment).

Although this embodiment will exemplify a case wherein a normalphotographing lens is used, it suffices to detect a foreign substance bymounting the above light unit on the lens mount 202. As described above,an image for foreign substance detection in this embodiment is an imagehaving a uniform color.

When the user issues an instruction to start foreign substance detectionprocessing by operating, for example, the 4-way selector switch 116 uponcompletion of the preparation, the microcomputer 402 performs aperturesetting first. The image forming state of a foreign substance near theimage sensor changes depending on the aperture value of the lens, andthe position of the foreign substance changes depending on the pupilposition of the lens. It is therefore necessary to hold an aperturevalue and the pupil position of the lens at the time of capturing of animage for foreign substance detection in foreign substance correctiondata in addition to the position and size of the foreign substance.

If, however, it is determined in advance in the step of generatingforeign substance correction data that the same aperture value is alwaysused even when a different lens is used, it is not always necessary tohold the aperture value in foreign substance correction data. Withregard to a pupil position as well, using a light unit or permitting theuse of only a specific lens will eliminate the necessity to hold a pupilposition in foreign substance correction data. That is, when the use ofa plurality of lenses is to be permitted or an aperture value to bestopped down is to be changed as needed in the step of generatingforeign substance correction data, it is necessary to hold an aperturevalue and the pupil position of the lens at the time of detection inforeign substance correction data. Note that the pupil position in thiscase is the distance from the image capturing plane (focal plane) of theexit pupil.

In this case, for example, F16 is designated (step S21).

The microcomputer 402 then causes the lens control circuit 407 toperform aperture vane control on the photographing lens 200 to set thestop to the aperture value designated in step S21 (step S22). Themicrocomputer 402 also sets the focus position to the infinity (stepS23).

Upon setting the aperture value and focus position of the photographinglens, the microcomputer 402 executes image capturing in the foreignsubstance detection mode (step S24). The capturing processing routine tobe performed in step S24 will be described in detail later withreference to FIG. 9. The image buffer memory 424 stores captured imagedata.

When image capturing is complete, the microcomputer 402 acquires theaperture value and lens pupil position at the time of image capturing(step S25). The microcomputer 402 reads out data corresponding to eachpixel of a captured image stored in the image buffer memory 424 into theimage processing circuit 425 (step S26). The image processing circuit425 performs the processing shown in FIG. 6 and acquires the positionand size of a pixel in which a foreign substance exists (step S27). Themicrocomputer 402 registers, in the foreign substance position memory427, the position and size of the pixel in which the foreign substanceexists, which are acquired in step S27, and the aperture value and lenspupil position information acquired in step S25 (step S28). In thiscase, when the above light unit is used, the microcomputer 402 cannotacquire lens information. If, therefore, the microcomputer 402 cannotacquire any lens information, the microcomputer 402 determines that thelight unit is used, and registers predetermined lens pupil positioninformation and the converted aperture value calculated from thediameter of the light source for the light unit.

In step S28, the microcomputer 402 checks the presence/absence of apixel defect by comparing the position of a fault pixel (pixel defect)at the time of the manufacture which is recorded in advance on the pixeldefect position memory 426 with the position of the readout pixel data.The position of only a foreign substance region for which it isdetermined that a pixel defect is irrelevant is allowed to be registeredin the foreign substance position memory 427.

FIG. 5 shows an example of the data format of foreign substancecorrection data stored in the foreign substance position memory 427. Asshown in FIG. 5, foreign substance correction data to be stored includeslens information and the position information and size information of aforeign substance at the time of capturing of a detection image. Thisforeign substance correction data is added to the image data togetherwith the capturing information of the image data at the time of normalimage capturing, and the resultant data is used for foreign substanceremoving processing to be described later.

More specifically, the microcomputer 402 stores the actual aperturevalue (f-number) at the time of capturing of a detection image and alens pupil position at that time as lens information at the time ofcapturing of a detection image. Subsequently, the microcomputer 402repeatedly stores parameters for each concrete foreign substance regionby the number of times corresponding to the number of foreign substanceregions. Parameters for each foreign substance region are a set of threenumerical values including the radius (e.g., two bytes) of a foreignsubstance, the x-coordinate (e.g., two bytes) of the center of aneffective image region, and the y-coordinate (e.g., two bytes) of thecenter.

If the size or the like of the foreign substance position memory 427imposes a restriction on the size of foreign substance correction data,the microcomputer 402 preferentially stores data starting from the headof the foreign substance region obtained in step S27. This is because inthe foreign substance region acquisition routine in step S27, foreignsubstance regions are sorted in the order of lesser noticeable foreignsubstances.

(Foreign Substance Region Acquisition Routine)

The foreign substance region acquisition routine to be performed in stepS27 will be described in detail next with reference to FIGS. 6 to 8.

As shown in FIG. 7, the microcomputer 402 rasterizes the readout imagedata in the memory, and processes the data on a block basis. Themicrocomputer 402 performs this processing to cope with vignetting dueto lens or sensor characteristics. Vignetting is a phenomenon in whichthe peripheral portion of a lens becomes lower in luminance than thecentral portion. It is known that reducing the aperture value of thelens will suppress this phenomenon to some extent. Even in a statewherein the lens is stopped down, determining a foreign substanceposition with respect to a captured image by using a predeterminedthreshold makes it impossible to accurately detect a foreign substanceon a peripheral portion depending on the lens to be used. For thisreason, the influence of vignetting is reduced by dividing an image intoblocks.

If, however, an image is simply divided into blocks, and differentthresholds are set for adjacent blocks, detection results across theblocks differ from each other. For this reason, adjacent blocks are madeto overlap, and a pixel determined as a foreign substance in either ofthe blocks constituting the overlap region is handled as a foreignsubstance region.

The microcomputer 402 performs foreign substance region determinationwithin each block in accordance with the processing procedure shown inFIG. 6. First of all, the microcomputer 402 calculates a maximumluminance Lmax and an average luminance Lave within a block andcalculates a threshold T1 within the block by using the followingequation:

T1=Lave×0.6+Lmax×0.4

The microcomputer 402 regards a pixel which does not exceed thethreshold as a foreign substance pixel (step S61), and regards anisolated region comprising the foreign substance pixel as a foreignsubstance region di (i=0, 1, . . . , n) (step S62). As shown in FIG. 8,the microcomputer 402 obtains, for each foreign substance region,maximum and minimum values Xmax and Xmin of the coordinates of the pixelforming the foreign substance region in the horizontal direction andmaximum and minimum values Ymax and Ymin of the coordinates in thevertical direction, thereby calculating a radius ri representing thesize of the foreign substance region di according to the followingequation (step S63):

ri=√[{(Xmax−Xmin)/2}²+{(Ymax−Ymin)/2}²]

FIG. 8 shows the relationship between Xmax, Xmin, Ymax, Ymin, and ri.

Subsequently, in step S64, the microcomputer 402 calculates an averageluminance value for each foreign substance region.

In some case, some restriction is imposed on the data size of foreignsubstance correction data due to a restriction on the size of theforeign substance position memory 427. In order to cope with such acase, pieces of foreign substance position information are sortedaccording to magnitudes and the average luminance values of foreignsubstance regions (step S65). In this embodiment, the microcomputer 402performs sorting in descending order of ri. If the pieces of informationare equal in ri, the microcomputer 402 performs sorting in descendingorder of average luminance values. This makes it possible topreferentially register noticeable foreign substances in foreignsubstance correction data. Let Di be a foreign substance region whichhas undergone sorting, and Ri be the radius of the foreign substanceregion Di.

If there is a foreign substance region larger than a predetermined size,it suffices to exclude it from sorting targets and place it at the endof a sorted foreign substance region list. This is because subsequentinterpolation processing for a large foreign substance region sometimesleads to degradation of image quality, and hence it is preferable tohandle such a region as the lowest in the order of preference of editingtargets.

(Capturing Processing Routine)

The capturing processing routine performed in step S24 in FIG. 4 will bedescribed next in detail with reference to the flowchart of FIG. 9. Themicrocomputer 402 executes this processing by executing the capturingprocessing program stored in the memory 428.

When executing this capturing processing routine, the microcomputer 402moves the quick return mirror 203 shown in FIG. 3 in step S201 toperform so-called mirror-up operation so as to make the quick returnmirror 203 retreat outside the capturing optical path.

In step S202, the microcomputer 402 starts accumulating electric chargesby using the image sensor. In step S203, the microcomputer 402 performsexposure by making the front and rear curtains 210 and 209 of theshutter shown in FIG. 3 travel. In step S204, the focusing screen 204completes the electric charge accumulation by the image sensor. In stepS205, the microcomputer 402 reads out an image signal from the imagesensor and temporarily stores the image data processed by the A/Dconverter 423 and the image processing circuit 425 in the image buffermemory 424.

Upon completing the readout of all image signals from the image sensorin step S206, the microcomputer 402 performs mirror-down operation ofthe quick return mirror 203 to return the quick return mirror to theoblique position, and terminates the series of capturing operations instep S207.

In step S208, the microcomputer 402 determines whether to perform normalimage capturing or foreign substance detection image capturing. When themicrocomputer 402 determines to perform normal image capturing, theprocess advances to step S209 to record camera setting values and thelike at the time of image capturing and the foreign substance correctiondata shown in FIG. 5 on the external storage device 419 in associationwith image data.

More specifically, the microcomputer 402 can associate the above databy, for example, adding foreign substance correction data to an Exifarea which is the header area of an image file on which camera settingvalues and the like at the time of image capturing are recorded.Alternatively, the microcomputer 402 can associate the data byindependently recording foreign substance correction data as a file andrecording only link information to the foreign substance correction datafile on the image data. If, however, the microcomputer 402 separatelyrecords the image file and the foreign substance correction data file,the link relationship may be lost upon movement of the image file. It istherefore preferable to integrally hold the foreign substance correctiondata and the image data.

(Foreign Substance Removing Processing)

A foreign substance removing processing procedure will be describednext. Foreign substance removing processing is performed on a separatelyprepared image processing apparatus instead of the digital camera body.

FIG. 10 is a view schematically showing the system configuration of theimage processing apparatus.

A CPU 1001 controls the overall operation of the system, and executesprograms stored in a primary storage unit 1002. The primary storage unit1002 is mainly a memory which reads and stores programs and the likestored in a secondary storage unit 1003. The secondary storage unit 1003corresponds to, for example, a hard disk. In general, a primary storageunit is smaller in capacity than a secondary storage unit. The secondarystorage unit stores programs, data, and the like which the primarystorage unit cannot afford to store. The secondary storage unit alsostores data which needs to be stored for a long period of time. In thisembodiment, the secondary storage unit 1003 stores programs, and whenexecuting a program, the CPU 1001 loads the program into the primarystorage unit 1002 and executes processing.

An input device 1004 corresponds to, for example, a card reader, ascanner, a film scanner, or the like which is required to input imagedata, as well as a mouse and a keyboard which are used to control thesystem. Examples of an output device 1005 are a monitor, a printer, andthe like. The method of arranging this apparatus may take various forms.However, they do not relate to the gist of the present invention, andhence a description will be omitted. Assume that this embodiment uses agenerally used mouse having left and right buttons as an input device.

The image processing apparatus is equipped with an operating systemcapable of parallel execution of a plurality of programs. The operatorcan operate a program operating on the image processing apparatus byusing a GUI (Graphical User Interface).

The image processing apparatus in this embodiment can execute twoprocesses as image editing processes. That is, one is a copy stampprocess and the other is a repair process. The copy stamp process is afunction of copying a partial region on a designated image onto anotherregion which is designated separately. The repair process is theprocessing of detecting an isolated region matching a predeterminedcondition within a designated region and interpolating the isolatedregion with surrounding pixels.

In addition, the image processing apparatus has an automatic repairfunction of automatically executing a repair process for designatedcoordinates by using foreign substance correction data attached to imagedata in the digital camera body. These processes will be described indetail later.

FIG. 11 is a view showing a GUI (Graphical User Interface) for an imageediting program in the image processing apparatus. The window includes aclose button 1100 and a title bar 1101. When the user presses the closebutton, he/she can terminate the program. When the user designates anddetermines an editing target image by dragging and dropping it in animage display region 1102, the CPU 1001 displays a file name on thetitle bar 1101 and then Fit-displays the target image in the imagedisplay region 1102.

The display state of an editing target image takes two modes, that is, aFit display mode and a pixel one-to-one display mode, which can beswitched by a display mode button 1108. According to this GUI, the userdesignates a modification position by clicking a corresponding portionon an image. In the Fit display mode, the CPU 1001 calculatescoordinates on a modification image which correspond to the clickedposition in accordance with a display magnification, and appliesprocessing to the coordinates. With this GUI, the user designates aprocessing range with a radius, which is a radius on the editing targetimage. This radius sometimes differs from the radius on the imagedisplay in the Fit display mode depending on the display magnification.

When the user presses an automatic repair process execution button 1103,the CPU 1001 executes automatic foreign substance removing processing tobe described later, and displays the processed image in the imagedisplay region 1102. The automatic repair process execution button 1103is valid only when an image has not been edited, and is invalid when theimage has been edited by the execution of a copy stamp process, a repairprocess, or an automatic repair process.

A radius slider 1106 is a slider by which the user designatesapplication ranges for a copy stamp process and a repair process.

When the user presses a repair process mode button 1104, the repairprocess mode is set. When the user left-clicks on an image in the repairprocess mode, the CPU 1001 applies a repair process (to be describedlater) to a region centered on the left-clicked coordinates and has aradius corresponding to the number of pixels designated by the radiusslider 1106. After the repair process is applied to the region, therepair process mode is canceled. When the user right-clicks on the imagedisplay region 1102 in the repair mode or presses any one of the buttonson the GUI, the repair mode is canceled.

When the user presses a copy stamp processing mode button 1105, the copystamp mode is set. When the user left-clicks on an image in the copystamp mode, the CPU 1001 sets the left-clicked coordinates as thecentral coordinates of a copy source region. When the user left-clickson the image while the central coordinates of the copy source region areset, the CPU 1001 executes a copy stamp process upon setting theleft-clicked coordinates as the central coordinates of the copydestination region and the radius designated by the radius slider 1106at this time point as a copy radius. The CPU 1001 then cancels the copystamp mode upon setting a state wherein the central coordinates of acopy source region have not been set. When the user right-clicks on theimage display region 1102 in the copy stamp mode or presses any one ofthe buttons on the GUI, the CPU 1001 cancels the copy stamp mode uponsetting a state wherein the central coordinates of a copy source regionhave not been set.

When the user presses a save button 1107, the CPU 1001 saves a processedimage.

According to the image editing program in this embodiment, as shown inFIG. 12, the CPU 1001 holds both an original image and an image afterprocesses. The editing process designated by the GUI and used forediting of the image is registered in an editing log. One editingprocess registered in the editing log will be referred to as an editingentry.

FIG. 13 shows an example of en editing entry.

An editing entry in this embodiment holds a process ID fordiscriminating a copy stamp process or a repair process, a center and aradius which indicate a process application region, relative coordinatesfrom copy source coordinates to copy destination coordinates which arerequired for a copy stamp process, and differential image data (to bedescribed later). When executing an automatic repair process, the CPU1001 executes a repair process in accordance with foreign substancecorrection data, and adds an editing entry to the editing log for everyexecution of a repair process.

Implementing these processes in this manner makes it possible toreconstruct an original image upon completely discarding an editing logand to cancel the immediately preceding editing process.

For example, it is possible to implement the processing of canceling animmediately preceding editing process by overwriting an temporarilyprocessed image with an original image or re-executing the editingprocess up to an editing entry immediately before an editing entry as acancellation target. If, however, the number of entries is very large,it takes much time to re-execute an editing process. For this reason,for every execution of editing operation, the CPU 1001 calculates thedifference between image data before and after the execution of anediting process, and holds it in an editing entry. Holding adifferential image allows return to the immediately preceding processedimage by only reflecting the differential image corresponding to thecancellation target editing entry in the processed image instead ofre-executing the editing process written in the editing entry from thebeginning.

Each of a repair process and an automatic repair process will bedescribed in detail next. Since a copy stamp process is a well knowntechnique, a detailed description of the technique will be omitted.

A repair process is the processing of detecting an isolated regionwithin a designated region and interpolating the isolated region. TheCPU 1001 implements a repair process by applying an interpolationroutine (to be described later) to a region expressed by the centralcoordinates and radius designated with the GUI.

In an automatic repair process, the CPU 1001 extracts foreign substancecorrection data from normally captured image data and automaticallyexecutes a repair process in accordance with foreign substancecorrection data. FIG. 14 shows a basic processing procedure for anautomatic repair process.

First of all, the CPU 1001 loads normally captured image data to whichforeign substance correction data is attached from the external storagedevice 419 in the digital camera or detached therefrom into the imageprocessing apparatus and stores the data in the primary storage unit1002 or the secondary storage unit 1003 (step S90).

The CPU 1001 then extracts the foreign substance correction dataattached to the captured image from the normally captured image data(the image subjected to a foreign substance removing processing) in stepS209 (step S91).

The CPU 1001 obtains a coordinate sequence Di (i=1, 2, . . . , n), aradius sequence Ri (i=1, 2, . . . , n), an aperture value f1, and a lenspupil position L1 from the foreign substance correction data extractedin step S91 (step S92). In this case, Ri represents the size of aforeign substance at the coordinates Di calculated in step S65 in FIG.6.

In step S93, the CPU 1001 acquires an aperture value f2 and a lens pupilposition L2 when the image has been normally captured, and converts Diaccording to the following equation in step S94. Let d be the distancefrom the center of the image to the coordinates Di and H be the distancefrom the surface of the image sensor 418 to a foreign substance.Coordinates Di′ after conversion and a radius Ri′ after conversion aregiven below:

Di′(x,y)=(L2×(L1−H)×d/((L2−H)×L1))×Di(x,y)

Ri′=(Ri×f1/f2+3)×2  (1)

In this case, the unit is a pixel, and “+3” concerning Ri′ represents amargin amount. The reason why (Ri×f1/f2+3) is doubled is that in orderto detect a foreign substance region by using an average luminance, aregion outside the foreign substance region is required.

In step S95, the CPU 1001 initializes an interpolation processingcounter i to 0. In step S96, the CPU 1001 counts up i.

In step S97, the CPU 1001 executes an interpolation routine (to bedescribed later) with respect to the region represented by the ithcoordinates Di′ and radius Ri′ to remove a foreign substance in theregion. In step S98, the CPU 1001 determines whether it has applied theforeign substance removing processing to all the coordinates. If theprocessing is complete for all the coordinates, the CPU 1001 terminatesthe processing. Otherwise, the process returns to step S96.

It is known that as the f-number at the time of image capturingdecreases (approaches the full open aperture value), the foreignsubstance image becomes blurred and less noticeable. It is thereforeconceivable to refer to the f-number at the time of image capturingbefore an automatic repair process and inhibit execution of all repairprocesses if this value is less than a threshold. This makes it possibleto omit an analysis process and the like, thereby efficiently processingeven a large number of editing target images. In addition, this canprevent degradation of image quality when an automatic repair process isperformed for even an unnoticeable foreign substance. For example, thisembodiment skips the above process when the f-number is less than f8 atwhich a foreign substance becomes unnoticeable.

FIG. 15 shows an automatic repair process procedure modified in thismanner.

This processing is the same as that shown in FIG. 14 except that the CPU1001 skips an interpolation routine (step S138) when an aperture valueat the time of image capturing, which is obtained by acquiringparameters at the time of image capturing before the execution of allprocesses, is compared with a threshold, and the aperture value is lessthan the threshold.

(Interpolation Routine)

An interpolation routine to be executed in a repair process andautomatic repair process will be described below.

FIG. 16 is a flowchart showing an interpolation routine procedure. Firstof all, in step S1201, the CPU 1001 performs foreign substance regiondetermination. Let P be the central coordinates of a region as a targetfor a repair process, and let R be a radius. Assume that a foreignsubstance region is a region satisfying all the following conditions:

(1) a region which is darker than a threshold T2 obtained by using anaverage luminance Yave and a maximum luminance Ymax of pixels containedin a repair process target region:

T2=Yave×0.6+Ymax×0.4

(2) a region which is not in contact with a circle represented by theabove central coordinates P and radius R; and(3) a region whose radius value calculated by the same method as that instep S63 in FIG. 6 is equal to or more than 11 pixels and less than 12pixels, with respect to an isolated region comprising a pixel lower inluminance than that selected in (1).

In an automatic repair process, a region which satisfies condition (4)in addition to the above conditions is a foreign substance region:

(4) a region which includes central coordinates P of a circle.

In this embodiment, 11 represents three pixels, and 12 represents 30pixels. This makes it possible to handle only an isolated small regionas a foreign substance.

If the CPU 1001 determines in step S1202 that there is such a region,the process advances to step S1203 to perform foreign substance regioninterpolation. Otherwise, the CPU 1001 terminates the processing. TheCPU 1001 executes the foreign substance region interpolation processingin step S1203 by a known defect region interpolation method. Forexample, as a known defect region interpolation method, the patternreplacement disclosed in Japanese Patent Laid-Open No. 2001-223894 isavailable. The technique disclosed in Japanese Patent Laid-Open No.2001-223894 specifies a defect region by using infrared light. Incontrast, this embodiment handles the foreign substance region detectedin step S1201 as a defect region, and interpolates the foreign substancewith surrounding normal pixels by pattern replacement. With regard to apixel which cannot be interpolated by pattern replacement, thecorresponding image data after pattern replacement is interpolated byusing the average color of p normal pixels selected in descending orderof distance from the interpolation target pixel and of q normal pixelsselected in ascending order of distance.

As described above, attaching foreign substance correction data to animage leads to the merit of eliminating the necessity to be conscious ofthe correspondence between foreign substance correction image data andcaptured image data. In addition, since foreign substance correctiondata is compact data comprising a position, a size, and conversion data(an aperture value and distance information concerning the pupilposition of the lens), the captured image data size will not beextremely large. Furthermore, performing interpolation processing foronly a region containing a pixel designated by foreign substancecorrection data makes it possible to greatly reduce the probability oferroneous detection. Moreover, controlling the execution/non-executionof an automatic repair process in accordance with the f-number at thetime of image capturing can perform more suitable processing.

Second Embodiment

In an automatic repair process in the first embodiment, it is simplydetermined from the f-number at the time of image capturing whether toapply foreign substance removing processing. In practice, however, as aforeign substance becomes larger, the substance becomes more noticeable,and hence an automatic repair process can be performed more properly bydetermining from a combination of an f-number and the size of a foreignsubstance whether to apply foreign substance removing processing,although the execution time cannot be shortened much.

FIG. 17 shows an automatic repair process procedure in this embodiment.A rough processing procedure in FIG. 17 is the same as that shown inFIG. 14 except that the processing of determining whether to execute aninterpolation routine is set before the execution of the interpolationroutine. In the execution determination processing in step S1707, forexample, the CPU determines whether to execute an interpolation routinedepending on whether the following inequality is satisfied:

β>√[f2]/(√[f1]×Ri)  (2)

where f1 is the f-number at the time of image capturing, f2 is thef-number at the time of capturing a foreign substance correction image,Ri is the radius of the ith foreign substance, and β is a predeterminedvalue.

Assume that in this embodiment, β is 10. In step S1707, the CPUdetermines whether inequality (1) is satisfied. If YES in step S1707,the process advances to step S1708 to execute interpolation processing.Otherwise, the process skips interpolation processing.

Note that the determination in step S1707 has the following meaning. Theright-hand side of inequality (1) decreases as the f-number at the timeof image capturing increases (the stop is stopped down) and the radiusRi of a foreign substance increases. That is, the right-hand side ofinequality (1) becomes smaller than a predetermined value of β when thestop is stopped down at the time of image capturing and a foreignsubstance is large in size and noticeable. As a consequence, theinterpolation processing in step S1708 is executed. That is, thedetermination in step S1707 indicates that only when a foreign substanceis noticeable, interpolation processing for the foreign substance regionis performed.

Execution of interpolation processing in accordance with conditions inthis manner can allow obtaining a more suitable interpolation result.This embodiment focuses on only the radius value of a foreign substanceregion in addition to an aperture value. However, it suffices tocalculate the luminance difference between a foreign substance regionand its neighboring region upon detection of a foreign substance, storeit in foreign substance correction data, and use it for executiondetermination processing in an interpolation routine. In some cases,when a thin foreign substance is captured with an aperture value closeto the full open aperture value, the foreign substance may not benoticeable. In this case, for example, it is conceivable to skip theprocessing.

Third Embodiment

According to the automatic repair process described in the firstembodiment, when an automatic repair process is executed, an attempt ismade to repair all designated coordinates. If, however, a designatedregion having a complicated pattern inside is interpolated, theinterpolation result may look unnatural. A designated region having acomplicated pattern may be a region with a lawn. This embodimenttherefore determines at the time of execution of an automatic repairprocess whether the inside of a designated region comprises pixels whichare uniform to some extent, and executes a repair process for only aregion which is uniform to some extent. A uniform region includes, forexample, a sky or a plain wall surface in a scenery image.

As shown in FIG. 18, therefore, before executing the interpolationroutine in step S1809, the CPU calculates a uniformity determinationvalue of a region other than a foreign substance region in step S1807.In step S1808, the CPU determines whether this value is equal to or lessthan a threshold a. Only when the value is equal, to or less than thethreshold a, the CPU executes a repair process according to theinterpolation routine in step S1809.

In this case, the threshold α is a threshold with respect to the averagevalue of amounts of change in luminance due to filter processing (to bedescribed later). Applying filter processing (to be described) to aregion will blur an isolated point or strong edge portion due to noiseor the like. In contrast, even if filter processing is applied to a flatportion, the luminance does not change much. Therefore, the calculatedluminance difference between regions before and after the application offilter processing can be used to determine whether the region ofinterest is flat. If the luminance difference between the regions beforeand after the application of the filter processing is smaller than thethreshold a, the CPU determines that the region is flat. In thisembodiment, if pixel values fall within the eight-bit accuracy, thethreshold α is, for example, six.

FIG. 19 is a flowchart showing a procedure for uniformity determinationvalue calculation processing in this embodiment.

In step S1901, the CPU obtains bitmap data comprising luminancecomponents after the application of a filter by applying the filtershown in FIG. 20A to the luminance components of the respective pixelswithin a designated region. The CPU calculates a luminance componentaccording to the following equation:

Y=0.299×R+0.587×G+0.114×B  (3)

Assume that the filter in FIG. 20A is a smoothing filter, and the CPUobtains a new pixel value for a pixel of interest by uniformly weightingnine pixels, that is, the upper left pixel, upper pixel, upper rightpixel, left pixel, pixel of interest, right pixel, lower left pixel,lower pixel, and lower right pixel, and averaging the sum of pixelvalues.

In step S1902, the CPU calculates the variance of luminance values ofpixels, of the bitmap data generated in step S1901, which are notcontained in the foreign substance region, after the application of thefilter, and uses the variance as a uniformity determination value forthe region.

Obviously, it suffices to perform filter processing by usingcoefficients other than those shown in FIG. 20A as long as a smoothingfilter for luminance components is used.

Performing such processing makes it possible to perform properinterpolation processing by extracting a uniform region while theinfluence of noise components of a captured image which occurs when theset ISO sensitivity is high is eliminated to some extent.

Fourth Embodiment

The third embodiment eliminates the influence of noise in the case of ahigh ISO sensitivity by using filter processing. In order to furthersuppress the influence of noise, as described in this embodiment, itsuffices to prepare a plurality of smoothing filters for luminancecomponents and calculate a uniformity determination value by obtainingthe difference between luminance bitmap data obtained by the filters.

This embodiment will exemplify uniformity determination valuecalculation processing using two kinds of smoothing filters. FIG. 21shows a flowchart for uniformity determination value calculationprocessing.

In step S2101, the CPU applies the filter shown in FIG. 20A to theluminance components of the respective pixels in a designated region andobtains first bitmap data comprising luminance components after theapplication of the filter.

In step S2102, the CPU applies the filter shown in FIG. 20B to theluminance components of the respective pixels in a designated region andobtains second bitmap data comprising luminance components after theapplication of the filter. Note that the filter in FIG. 20B is used touniformly weight 25 pixels including a pixel of interest and neighboringpixels and obtain a new pixel value for the pixel of interest byaveraging the sum of the pixel values.

In step S2103, the CPU calculates the average value of differencesbetween corresponding pixels of the first and second bitmap data, andsets it as a uniformity determination value.

Performing such processing makes it possible to perform properinterpolation processing by extracting a uniform region while theinfluence of noise components on a captured image is eliminated.

Fifth Embodiment

The third and fourth embodiments have exemplified the case wherein whenthe uniformity of a target region is to be determined, filter processingis applied to the entire target region.

The methods according to the third and fourth embodiments use smoothingfilters, and hence cannot sometimes accurately determine uniformity withrespect to an image with large luminance noise. This embodimenttherefore divides a designated region into blocks, and obtains auniformity determination value for each block, thereby improving theaccuracy of detection of a uniform region.

FIG. 22 is a flowchart showing a procedure for uniformity determinationvalue calculation processing in this embodiment.

First of all, in step S2201, the CPU divides a designated region intoblocks each having a predetermined size. For example, in thisembodiment, the block size is set to eight pixels in the verticaldirection and eight pixels in the horizontal direction.

In step S2202, the CPU obtains the variance of effective pixel luminancevalues with respect to each block containing effective pixels equal toor more than μ pixels. In this case, an effective pixel is a pixel whichis not contained in a foreign substance region, and is contained in aregion designated as a repair process target. Assume that in thisembodiment, μ is 32.

Finally, in step S2203, the CPU calculates the variance of uniformitydetermination values for the respective blocks, and sets the variance asa uniformity determination value for the region designated as a repairprocess target.

Performing such processing makes it possible to obtain a region withhigh uniformity regardless of the amount of noise.

It is further conceivable to obtain uniformity determination values forthe respective blocks and calculate the variance of them, as shown inFIG. 23. The methods according to the third and fourth embodiments maydetermine even a region containing a portion which is locallynon-uniform as a uniform region. However, using the method of the fifthembodiment can solve such a problem. The scheme shown in FIG. 22determines even a region with a large amount of noise as a uniformregion as long as it is uniform as a whole. In contrast, the schemeshown in FIG. 23 can discriminate only a portion whose noise amount issmall to some extent.

Sixth Embodiment

When an automatic repair process is executed according to the imageediting program in the first embodiment, part of an application resultmay be an inappropriate interpolation result. The sixth embodiment willexemplify a method of canceling an editing process when part ofinterpolation is inappropriate.

An image editing program of this embodiment holds an original image andan entire editing log, as described in the first embodiment. Therefore,as shown in FIG. 24, an editing entry corresponding to editing contentsto be canceled is deleted from an editing log, and an image afterprocessing is generated from the original image and the editing log fromwhich the editing entry to be canceled is deleted. FIG. 25 shows aflowchart for editing content cancellation processing.

In step S2501, the CPU copies the original image on an image afterprocessing, and initializes the image after processing. In step S2502,the CPU deletes a target editing entry from an editing log. Finally, instep S2503, the CPU executes editing processing recorded in a newediting log with respect to the initialized image.

Executing this processing makes it possible to obtain image data in astate wherein only the editing contents corresponding to the deletedentry are canceled. Subsequently, canceling editing contents by deletinga specific editing entry will be simply expressed as “canceling editingprocessing”.

FIG. 26 shows an example of a cancellation designation dialog fordesignating the cancellation of editing processing.

When the user performs a specific key operation (e.g., Ctrl+D) in animage editing program, the CPU displays a cancellation designationdialog as a modal dialog. An editing log display unit 3100 displaysediting entries in the form of a list in the order in which they areregistered in an editing log. The list displays serial numbers andediting contents (a repair process or a copy stamp process). Note thatthe editing log display unit 3100 is displayed on part of an imagedisplay region 1102 shown in FIG. 11. When the user selects an editingentry display on the list, the CPU displays the corresponding editingportion on the image display region 1102 upon circling it to present theuser which part is selected. When the user presses a cancellationdesignation button 3101, the CPU cancels editing processingcorresponding to the selected editing entry. When the user presses awindow erase button 3102, the CPU closes the dialog.

However, a problem arises when entire editing processing isunconditionally canceled.

An image editing program in this embodiment comprises a function ofperforming a copy stamp process. If editing processing is canceled whilea region for which editing processing is canceled is designated as acopy source region for a copy stamp process, image data before theremoval of a foreign substance is copied to a copy destination region.In addition, the copy destination region may also be designated as acopy source for another copy stamp process.

FIGS. 27A to 27D are views for explaining this problem.

Assume that there are two foreign substances in an initial state. Assumethat these foreign substances are the black regions in FIG. 27A. Asshown in FIG. 27B, a repair process is performed for one of theseregions. As is obvious from FIGS. 27B and 27C, the regions havingundergone repair processes are indicated by the oblique lines.Subsequently, as shown in FIG. 27C, the other region is deleted by usingthis interpolation result. When an editing entry for the repair processperformed in FIG. 27B is deleted, no repair process is performed. Theforeign substance regions which have been repair process targets arecopied, as shown in FIG. 27D.

This embodiment therefore provides the processing of determining whetherto cancel editing processing. More specifically, if an editing logincludes the processing of referring to a pixel at the coordinatesdesignated in advance as in a copy stamp process, the cancellation ofediting processing is inhibited.

A repair process is a process which is complete within a region twice aslarge as a radius on an algorithm, and interpolation processing isexecuted by adaptively selecting pixels in accordance with the state ofa surrounding region. If, therefore, an editing log comprises only arepair process, cancellation of an arbitrary editing process poses noproblem.

FIG. 28 is a flowchart showing a procedure of cancellation inhibitiondetermination processing in editing processing in this embodiment.

In step S2701, the CPU initializes a cancellation flag to TRUE. In stepS2702, the CPU initializes a counter i. In step S2703, the CPU counts upthe counter i. In step S2704, the CPU checks whether the ith editingentry is a copy stamp process. If NO in step S2704, the process advancesto step S2705. If YES in step S2704, the process advances to step S2706.In step S2705, the CPU checks whether all the editing entries arechecked. If NO in step S2705, the process returns to step S2703.Otherwise, the CPU terminates the processing. In step S2706, since anediting log includes a copy stamp process, the CPU sets the cancellationflag to FALSE and terminates the processing. As a result of thisprocessing, the CPU permits the cancellation of editing processing onlywhen the cancellation flag is TRUE.

Assume that when the CPU checks all the editing entries for copy stampprocesses included in the editing log, not all the editing entries forthe copy stamp processes include portions modified by copy stampprocesses as copy source regions. In this case, it is possible toperform cancellation processing for editing contents. From the viewpointof the user of the image processing program, however, it is verycomplicated to determine whether to inhibit canceling operation for anediting entry, and hence it is very difficult to use such determinationprocessing. If, for example, an editing entry includes both a repairprocess and a copy stamp process, it is difficult to determine whetherthe editing state can be canceled, even if the editing log is displayedin the form of a list. The determination method in this embodimentinhibits the cancellation of an editing entry if a copy stamp processhas been executed at least once, and hence allows the user to easilycomprehend a cancellation inhibition condition for an editing entry.

Seventh Embodiment

This embodiment will exemplify the processing of updating foreignsubstance correction data upon cancellation of editing processing in thesixth embodiment.

In some case, when a user fixes a camera on a tripod to capture a stillobject in a studio, there are a plurality of cuts with almost the samecomposition. Assume that an automatic repair process is performed forone of such images, and some portion exhibits an inappropriateinterpolation result. In this case, the remaining images may have thesame result.

It is therefore conceivable to add a cancellation flag (e.g., one byte)indicating whether editing processing has been canceled to parametersfor a foreign substance region which are stored in foreign substancecorrection data and, if editing processing has been canceled, set thecancellation flag for the corresponding foreign substance region. In anautomatic repair process, the processing is skipped for a foreignsubstance region in which the cancellation flag is set. Introducing sucha mechanism makes it possible to shorten the operation time required forthe cancellation of editing processing by excluding in advance portionswhich may have inappropriate results from processing targets.

FIG. 29 shows an example of foreign substance correction data in thisembodiment. Note, however, that foreign substance correction data to berecorded on the camera body may remain in the state shown in FIG. 5. Inthis case, when foreign substance correction data to be executed by theimage editing program is acquired (e.g., step S91 in FIG. 14), the datamay be converted into the form shown in FIG. 29. Assume that the initialstate of the cancellation flag is FALSE.

FIG. 30 is a flowchart for an automatic repair process in thisembodiment. This process differs from that shown in FIG. 14 in that itchecks the cancellation flag in step S2907 before the execution of theinterpolation routine. If the cancellation flag is TRUE, interpolationprocessing (step S2908) is skipped.

FIG. 31 is a view for explaining processing upon cancellation of editingprocessing in this embodiment.

When the cancellation of editing contents is designated, a correspondingentry is deleted from an editing log. In addition, if a foreignsubstance corresponding to the deleted editing log information exists inforeign substance correction data, the cancellation flag in thecorresponding foreign substance region is set to TRUE. An editing entrybased on GUI operation may have been added to an editing log in additionto an editing entry based on an automatic repair process. Discriminationbetween them is performed by using a separately prepared correspondencetable (not shown) of editing entries and foreign substance regioninformation in foreign substance correction data. Alternatively, holdingthe serial numbers of foreign substance regions within editing entriesmakes it possible to specify foreign substance region informationcorresponding to each editing entry. In this case, designating “−1”instead of a serial number for an editing entry designated by using theGUI allows identification of the absence of corresponding foreignsubstance information.

Other Embodiments

The objects of the respective embodiments can also be achieved by thefollowing method. A storage medium (or a recording medium) storingsoftware program codes for implementing the functions of the aboveembodiments is supplied to a system or apparatus. The computer (or a CPUor MPU) of the system or apparatus then reads out and executes theprogram codes stored in the storage medium. In this case, the programcodes read out from the storage medium implement the functions of theabove embodiments by themselves, and the storage medium storing theprogram codes constitutes the present invention. The functions of theabove embodiments are implemented not only when the readout programcodes are executed by the computer but also when the operating system(OS) running on the computer performs part or all of actual processingbased on the instructions of the program codes.

In addition, the functions of the above embodiments are also implementedwhen the program codes read out from the storage medium are written inthe memory of a function expansion board inserted into the computer or afunction expansion unit connected to the computer, and the CPU of thefunction expansion board or function expansion unit performs part or allof actual processing based on the instructions of the program codes.

When the present invention is applied to the above storage medium, thestorage medium stores program codes corresponding to the proceduresdescribed above.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2006-203740 filed on Jul. 26, 2006, which is hereby incorporated byreference herein in its entirety.

1-4. (canceled)
 5. An image capturing apparatus comprising: an image capture unit configured to capture an object image, the image capture unit obtaining normal image data and foreign substance detection image data; a foreign substance detection unit configured to detect, from the foreign substance detection image data, foreign substance information that is information of a position and a size of a foreign substance in an image sensing plane of the image capture unit; a lens information acquisition unit configured to acquire lens information of a lens used upon capturing the object image; and a recording unit configured to record the normal image data obtained by capturing the object as a normal image data file, wherein in a case when said recording unit records the normal image data, (1) the foreign substance information, (2) the first lens information of a lens used upon capturing the foreign substance detection image, and (3) the second lens information of a lens used upon capturing the normal image are written in a header region of the normal image data file, whereby (1) the foreign substance information, (2) the first lens information of the lens used upon capturing the foreign substance detection image, and (3) the second lens information of the lens used upon capturing the normal image are recorded in association with the normal image data.
 6. A method of controlling an image capturing apparatus having an image capture unit configured to capture an object image, the image capture unit obtaining normal image data and foreign substance detection image data, comprising the steps of: detecting, from the foreign substance detection image data, foreign substance information that is information of a position and a size of a foreign substance in an image sensing plane of the image capture unit; acquiring lens information of a lens used upon capturing the object image; and recording the normal image data obtained by capturing the object as a normal image data file, wherein in a case when the normal image data is recorded, (1) the foreign substance information, (2) the first lens information of a lens used upon capturing the foreign substance detection image, and (3) the second lens information of a lens used upon capturing the normal image are written in a header region of the normal image data file, whereby (1) the foreign substance information, (2) the first lens information of the lens used upon capturing the foreign substance detection image, and (3) the second lens information of the lens used upon capturing the normal image are recorded in association with the normal image data.
 7. A non-transitory computer-readable medium for storing a program characterized by causing a computer to execute a method of controlling an image capturing apparatus having an image capture unit configured to capture an object image, the image capture unit obtaining normal image data and foreign substance detection image data, comprising the steps of: detecting, from the foreign substance detection image data, foreign substance information that is information of a position and a size of a foreign substance in an image sensing plane of the image capture unit; acquiring lens information of a lens used upon capturing the object image; and recording the normal image data obtained by capturing the object as a normal image data file, wherein in a case when the normal image data is recorded, (1) the foreign substance information, (2) the first lens information of a lens used upon capturing the foreign substance detection image, and (3) the second lens information of a lens used upon capturing the normal image are written in a header region of the normal image data file, whereby (1) the foreign substance information, (2) the first lens information of the lens used upon capturing the foreign substance detection image, and (3) the second lens information of the lens used upon capturing the normal image are recorded in association with the normal image data. 